Chardata docs by pp-mo · Pull Request #7194 · SciTools/iris

pp-mo · 2026-07-07T11:14:55Z

In writing this, I have encountered a bit of a doubt about the implementation : #7195
But for the sake of getting this under review, I'm going to treat that as a separate issue :
I believe it can only simplify the account here.

codecov · 2026-07-07T11:21:28Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.29%. Comparing base (ebc6531) to head (93720c1).

Additional details and impacted files

@@                Coverage Diff                @@
##           FEATURE_chardata    #7194   +/-   ##
=================================================
  Coverage             90.29%   90.29%           
=================================================
  Files                    92       92           
  Lines                 25171    25171           
  Branches               4700     4700           
=================================================
  Hits                  22728    22728           
  Misses                 1673     1673           
  Partials                770      770

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Changes to pre-commit actually occurred in the target branch 'FEATURE_chardata'.
Only appeared here because I temporarily changed the target to 'main' to get docs building

pt331

My review is all over the place since I was adding them as I was reading and rereading, but I think I understand the overall idea. Most of the comments are nitpicks or thoughts on structuring.

pt331 · 2026-07-09T13:07:44Z

+        encoding support (described below).
+
+
+In Iris


I think the Iris part of this section should be the first under summary, since I imagine users are most interested in that. The other sections are more to provide context for decisions made in Iris

pt331 · 2026-07-09T14:18:46Z

+    cubes and coordinates.
+
+    *   the numpy dtype is of the type "U<xx>", where <xx> is a character width.
+    *   the character width relates to a string *dimension* of the netCDF variable,


I'm not following what this bullet point means.

pt331 · 2026-07-09T14:43:51Z



+Character and String datatypes
+------------------------------


I think an introduction might be warranted. Something like:

In NetCDF, text can be represented in 2 ways:

char: the number of characters in the text is fixed

string: the number of characters in the text can vary

Then text can have different encodings, which affects the number of bytes used to store a character. For example:

ASCII and UTF-32: the number of bytes are fixed

UTF-8 and UTF-16: the number of bytes can vary

This section discusses how Iris handles the different cases and how the standards and interpretations have changed.

pp-mo · 2026-07-13T23:33:14Z

Hi again @pt331 !
I think I've responded to all your comments now.
The last 2 commits have mixed things up a bit, and moved parts around (partly as you suggested). And I made some significant changes there too. So, those commits don't make easy diffs, and may be worth considering separately
So, it's changed quite a lot, and probably wants re-considering as a whole now -- please feel free to add fresh suggestions.

pt331 · 2026-07-14T12:32:58Z

+determined by **inverting the above relations**.
+
+The actual maximum number of characters in the data will always be less than or equal to
+this, since the maximum possible character width is for all-ascii data.


I assume you meant to say

Suggested change

this, since the maximum possible character width is for all-ascii data.

this, since the maximum possible string width is 1 byte for all-ascii data.

Well, not exactly..
I meant that the longest string you can get is when the bytes are full with all ascii characters.
I'll try + rephrase this more clearly...

pt331 · 2026-07-14T12:37:23Z

+The width created by reading will always round-trip correctly, i.e. the dimension
+length is unchanged if data is read and then written back.
+
+Summary of Iris string handling


I think this summary can be removed now and once you have more concrete examples (mentioned in private), something like "Iris string handling in practice" would be a good replacement.

pt331 · 2026-07-14T13:00:09Z

+        See in netCDF4 python documentation :
+        `Dealing with strings <https://unidata.github.io/netcdf4-python/#dealing-with-strings>`_.
+        **However,** Iris turns this feature *off*, in order to implement its own
+        wider-ranging encoding support (described below).


no longer described below

pt331 · 2026-07-14T13:11:18Z

+adopted when translating to and from bytes, and rules for determining what the encoding
+is or was.
+
+Background


I think the Background can now be moved above "String data in NetCDF", "String data in the NetCDF CF Conventions" and "String data in the netCDF4 Python module" and make them subheadings

OK that's nice + a simpler way of doing it than creating a separate new page.

Co-authored-by: pt331 <144435193+pt331@users.noreply.github.com>

pt331

Looks good! I like the idea to link to other sections for further details so that the user can shortcut to them if they feel it's more appropriate for them to know. The only changes left are broken links

pt331 · 2026-07-14T15:54:47Z

+Character and String datatypes
+------------------------------
+Text can be present  in NetCDF in a variety of ways
+(see : `String data in NetCDF <cf_strings>`_ for details).


Link is broken. I think it should be

Suggested change

(see : `String data in NetCDF <cf_strings>`_ for details).

(see : `String data in NetCDF <cf-strings>`_ for details).

pt331 · 2026-07-14T15:59:47Z

+coordinate ``.points``.
+
+These are always stored as arrays of numpy dtype "U<xx>", where <xx> is a maximum
+string width (which could be either `real or lazy <real_and_lazy_data>`_).


Link is broken. I think it should be

Suggested change

string width (which could be either `real or lazy <real_and_lazy_data>`_).

string width (which could be either `real or lazy <real-and-lazy_data>`_).

This comment was marked as outdated.

Sign in to view

pp-mo changed the base branch from main to FEATURE_chardata July 7, 2026 11:15

pp-mo changed the base branch from FEATURE_chardata to main July 7, 2026 11:20

pp-mo added 6 commits July 7, 2026 17:03

Remove the DECODE_TO_STRINGS_ON_READ from public API.

7986f18

First-draft chardata docs.

b4f1a3f

Fix titles and re-order slightly.

6387a3e

Disable warn-to-fail.

e645ea1

Improves.

77ab73f

Pin mpl for docs build, reinstate warn-to-fail.

93ffd64

pp-mo force-pushed the chardata_docs branch from f19c4c4 to 93ffd64 Compare July 7, 2026 16:03

Fix bullets

a52e50d

scitools-ci Bot added this to 🚴 Peloton Jul 8, 2026

pp-mo changed the base branch from main to FEATURE_chardata July 8, 2026 10:19

pp-mo force-pushed the chardata_docs branch from a841539 to 92c5279 Compare July 8, 2026 11:18

More improvements

473db52

pp-mo force-pushed the chardata_docs branch from 92c5279 to 473db52 Compare July 8, 2026 12:16

pp-mo mentioned this pull request Jul 8, 2026

Allow saving non-ascii strings to netCDF with no '_Encoding' attribute #7195

Open

pp-mo requested a review from pt331 July 8, 2026 14:27

pp-mo linked an issue Jul 8, 2026 that may be closed by this pull request

Complete documentation of chardata handling #6924

Open

2 tasks

pp-mo marked this pull request as ready for review July 8, 2026 15:34

pt331 reviewed Jul 9, 2026

View reviewed changes

pp-mo added 4 commits July 13, 2026 12:41

Use plain 'code style' for netCDF char/string type names.

ff827f1

Add some references and fix links.

82e3588

Fix capitalisation of 'netcdf' in various contexts.

da24fe9

Various clarity improvements , including review requests.

dfe7714

pp-mo force-pushed the chardata_docs branch from 9855324 to dfe7714 Compare July 13, 2026 15:17

pp-mo added 2 commits July 13, 2026 18:45

Rewrites prior to re-ordering.

e7a2068

Re-order, with improved intro, as suggested in review.

ad76944

pt331 reviewed Jul 14, 2026

View reviewed changes

Comment thread docs/src/user_manual/explanation/netcdf_io.rst Outdated

Comment thread docs/src/user_manual/explanation/netcdf_io.rst Outdated

Comment thread docs/src/user_manual/explanation/netcdf_io.rst Outdated

pp-mo and others added 4 commits July 14, 2026 15:07

Update docs/src/user_manual/explanation/netcdf_io.rst

b6104c8

Co-authored-by: pt331 <144435193+pt331@users.noreply.github.com>

Update docs/src/user_manual/explanation/netcdf_io.rst

6bdc296

Co-authored-by: pt331 <144435193+pt331@users.noreply.github.com>

Update docs/src/user_manual/explanation/netcdf_io.rst

b14f778

Co-authored-by: pt331 <144435193+pt331@users.noreply.github.com>

Various review actions, and further small improvements.

93720c1

pt331 reviewed Jul 14, 2026

View reviewed changes



		Character and String datatypes
		------------------------------

	this, since the maximum possible character width is for all-ascii data.
	this, since the maximum possible string width is 1 byte for all-ascii data.

	(see : `String data in NetCDF <cf_strings>`_ for details).
	(see : `String data in NetCDF <cf-strings>`_ for details).

	string width (which could be either `real or lazy <real_and_lazy_data>`_).
	string width (which could be either `real or lazy <real-and-lazy_data>`_).

Uh oh!

Conversation

pp-mo commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

codecov Bot commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

pt331 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pp-mo commented Jul 13, 2026

Uh oh!

pt331 Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pt331 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

pp-mo commented Jul 7, 2026 •

edited

Loading

codecov Bot commented Jul 7, 2026 •

edited

Loading

pt331 left a comment •

edited

Loading

pt331 Jul 14, 2026 •

edited

Loading